{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FINN - End-to-End Flow\n",
    "-----------------------------------------------------------------\n",
    "\n",
    "In this notebook, we will show how to take a simple, binarized, fully-connected network trained on the MNIST data set and take it all the way down to a customized bitfile running on a PYNQ board. \n",
    "\n",
    "This notebook is quite lengthy, and some of the cells (involving Vivado synthesis) may take up to an hour to finish running. To let you save and resume your progress, we will save the intermediate ONNX models that are generated in the various steps to disk, so that you can jump back directly to where you left off.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "The FINN compiler comes with many *transformations* that modify the ONNX representation of the network according to certain patterns. This notebook will demonstrate a *possible* sequence of such transformations to take a particular trained network all the way down to hardware, as shown in the figure below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](finn-design-flow-example.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The white fields show the state of the network representation in the respective step. The colored fields represent the transformations that are applied to the network to achieve a certain result. The diagram is divided into 5 sections represented by a different color, each of it includes several flow steps. The flow starts in top left corner with Brevitas export (green section), followed by the preparation of the network (blue section) for the Vivado HLS synthesis and Vivado IPI stitching (orange section), and finally building a PYNQ overlay bitfile and testing it on a PYNQ board (yellow section).\n",
    "There is an additional section for functional verification (red section) on the right side of the diagram, which we will not cover in this notebook. For details please take a look in the verification notebook which you can find [here](tfc_end2end_verification.ipynb)\n",
    "\n",
    "\n",
    "This Jupyter notebook is organized based on the sections described above. We will use the following helper functions, `showSrc` to show source code of FINN library calls and `showInNetron` to show the ONNX model at the current transformation step. The Netron displays are interactive, but they only work when running the notebook actively and not on GitHub (i.e. if you are viewing this on GitHub you'll only see blank squares)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.util.visualization import showSrc, showInNetron\n",
    "from finn.util.basic import make_build_dir\n",
    "\n",
    "    \n",
    "build_dir = \"/workspace/finn\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "-------------\n",
    "1. [Brevitas export](#brev_exp)\n",
    "2. [Network preparation](#nw_prep)\n",
    "3. [Vivado HLS and IPI](#vivado)\n",
    "4. [PYNQ hardware generation and deployment](#hw_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Brevitas export <a id='brev_exp'></a>\n",
    "FINN expects an ONNX model as input. This can be a model trained with [Brevitas](https://github.com/Xilinx/brevitas). Brevitas is a PyTorch library for quantization-aware training and the FINN Docker image comes with several [example Brevitas networks](https://github.com/maltanar/brevitas_cnv_lfc). To show the FINN end-to-end flow, we'll use the TFC-w1a1 model as example network.\n",
    "\n",
    "First a few things have to be imported. Then the model can be loaded with the pretrained weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/workspace/brevitas_cnv_lfc/training_scripts/models/TFC.py:85: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  x = 2.0 * x - torch.tensor([1.0]).to(self.device)\n"
     ]
    }
   ],
   "source": [
    "import onnx\n",
    "from finn.util.test import get_test_model_trained\n",
    "import brevitas.onnx as bo\n",
    "\n",
    "tfc = get_test_model_trained(\"TFC\", 1, 1)\n",
    "bo.export_finn_onnx(tfc, (1, 1, 28, 28), build_dir+\"/tfc_w1_a1.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model was now exported, loaded with the pretrained weights and saved under the name \"lfc_w1_a1.onnx\".\n",
    "To visualize the exported model, Netron can be used. Netron is a visualizer for neural networks and allows interactive investigation of network properties. For example, you can click on the individual nodes and view the properties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Serving '/workspace/finn/tfc_w1_a1.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1ad0b6e80>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "showInNetron(build_dir+\"/tfc_w1_a1.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the model in .onnx format, we can work with it using FINN. For that FINN `ModelWrapper` is used. It is a wrapper around the ONNX model which provides several helper functions to make it easier to work with the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.core.modelwrapper import ModelWrapper\n",
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the model is prepared and could be simulated using Python. How this works is described in the Jupyter notebook about verification and can be found [here](tfc_end2end_verification.ipynb#simpy).\n",
    "\n",
    "The model can now also be processed in different ways. The principle of FINN are analysis and transformation passes, which can be applied to the model. An analysis pass extracts specific information about the model and returns it to the user in the form of a dictionary. A transformation pass changes the model and returns the changed model back to the FINN flow.\n",
    "\n",
    "Since the goal in this notebook is to process the model to such an extent that a bitstream can be generated from it, the focus is on the transformations that are necessary for this. In the next section these are discussed in more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Network preparation <a id='nw_prep'></a>\n",
    "\n",
    "* [FINN-style Dataflow Architectures](#dataflow_arch)\n",
    "* [Tidy-up transformations](#basic_trafo)\n",
    "* [Streamlining](#streamline)\n",
    "* [Conversion to HLS layers](#hls_layers)\n",
    "* [Creating a Dataflow Partition](#dataflow_partition)\n",
    "* [Folding and Datawidth Converter, FIFO and TLastMarker Insertion](#folding)\n",
    "\n",
    "\n",
    "In this section, we will put the network through a series of transformations that puts it in a form that can be stitched together to form a FINN-style dataflow architecture, yielding a high-performance, high-efficiency FPGA accelerator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FINN-style Dataflow Architectures <a id='dataflow_arch'></a>\n",
    "\n",
    "We start with a quick recap of FINN-style dataflow architectures. The key idea in such architectures is to parallelize across layers as well as within layers by dedicating a proportionate amount of compute resources to each layer, as illustrated in the figure below taken from the [FINN-R paper](https://arxiv.org/pdf/1809.04570.pdf):\n",
    "\n",
    "![](finn-hw-arch.png)\n",
    "\n",
    "In practice, the compute arrays are instantiated by function calls to optimized Vivado HLS building blocks from the [finn-hlslib](https://github.com/Xilinx/finn-hlslib) library. As these function calls can only handle certain patterns/cases, we need to transform the network into an appropriate form so that we can replace network layers with these function calls, which is the goal of the network preparation process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tidy-up transformations <a id='basic_trafo'></a>\n",
    "This section deals with some basic transformations, which are applied to the model like a kind of \"tidy-up\" to make it easier to be processed. They do not appear in the diagram above, but they are applied in many steps in the FINN flow to postprocess the model after a transformation and/or to prepare it for the next transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These transformations are:\n",
    "* GiveUniqueNodeNames\n",
    "* GiveReadableTensorNames\n",
    "* InferShapes\n",
    "* InferDataTypes\n",
    "* FoldConstants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first two transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames`) the nodes in the graph are first given unique (by enumeration) names, then the tensors are given human-readable names (based on the node names). The following two transformations (`InferShapes`, `InferDataTypes`) derive the shapes and data types of the tensors from the model properties and set them in the `ValueInfo` of the model. These transformations can almost always be applied without negative effects and do not affect the structure of the graph, ensuring that all the information needed is available.\n",
    "\n",
    "The last listed transformation is `FoldConstants`, which performs constant folding. It identifies a node with constant inputs and determines its output. The result is then set as constant-only inputs for the following node and the old node is removed. Although this transformation changes the structure of the model, it is a transformation that is usually always desired and can be applied to any model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These transformations can be imported and applied as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.general import GiveReadableTensorNames, GiveUniqueNodeNames\n",
    "from finn.transformation.infer_shapes import InferShapes\n",
    "from finn.transformation.infer_datatypes import InferDataTypes\n",
    "from finn.transformation.fold_constants import FoldConstants\n",
    "\n",
    "model = model.transform(InferShapes())\n",
    "model = model.transform(FoldConstants())\n",
    "model = model.transform(GiveUniqueNodeNames())\n",
    "model = model.transform(GiveReadableTensorNames())\n",
    "model = model.transform(InferDataTypes())\n",
    "\n",
    "model.save(build_dir+\"/tfc_w1_a1_tidy.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of these transformations can be viewed with netron after the model has been saved again. By clicking on the individual nodes, it can now be seen, for example, that each node has been given a name. Also the whole upper area could be folded, so that now the first node is \"Reshape\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_tidy.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1ad0639e8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "showInNetron(build_dir+\"/tfc_w1_a1_tidy.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Streamlining <a id='streamline'></a>\n",
    "Streamlining is a transformation containing several sub-transformations. The goal of streamlining is to eliminate floating point operations by moving them around, then collapsing them into one operation and in the last step transform them into multi-thresholding nodes. For more information on the theoretical background of this, see [this paper](https://arxiv.org/pdf/1709.04060).\n",
    "\n",
    "Let's have a look at which sub-transformations `Streamline` consists of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "class Streamline(Transformation):\n",
      "    \"\"\"Apply the streamlining transform, see arXiv:1709.04060.\"\"\"\n",
      "\n",
      "    def apply(self, model):\n",
      "        streamline_transformations = [\n",
      "            ConvertSubToAdd(),\n",
      "            ConvertDivToMul(),\n",
      "            BatchNormToAffine(),\n",
      "            ConvertSignToThres(),\n",
      "            MoveAddPastMul(),\n",
      "            MoveScalarAddPastMatMul(),\n",
      "            MoveScalarAddPastConv(),\n",
      "            MoveScalarMulPastMatMul(),\n",
      "            MoveScalarMulPastConv(),\n",
      "            MoveAddPastMul(),\n",
      "            CollapseRepeatedAdd(),\n",
      "            CollapseRepeatedMul(),\n",
      "            AbsorbAddIntoMultiThreshold(),\n",
      "            FactorOutMulSignMagnitude(),\n",
      "            AbsorbMulIntoMultiThreshold(),\n",
      "            Absorb1BitMulIntoMatMul(),\n",
      "            Absorb1BitMulIntoConv(),\n",
      "            RoundAndClipThresholds(),\n",
      "        ]\n",
      "        for trn in streamline_transformations:\n",
      "            model = model.transform(trn)\n",
      "            model = model.transform(GiveUniqueNodeNames())\n",
      "            model = model.transform(GiveReadableTensorNames())\n",
      "            model = model.transform(InferDataTypes())\n",
      "        return (model, False)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from finn.transformation.streamline import Streamline\n",
    "showSrc(Streamline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen, several transformations are involved in the streamlining transformation. There are move and collapse transformations. In the last step the operations are transformed into multithresholds. The involved transformations can be viewed in detail [here](https://github.com/Xilinx/finn/tree/master/src/finn/transformation/streamline). After each transformation, three of the tidy-up transformations (`GiveUniqueNodeNames`, `GiveReadableTensorNames` and `InferDataTypes`) are applied to the model.\n",
    "\n",
    "After streamlining the network looks as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_streamlined.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1346e4ef0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1_tidy.onnx\")\n",
    "model = model.transform(Streamline())\n",
    "model.save(build_dir+\"/tfc_w1_a1_streamlined.onnx\")\n",
    "showInNetron(build_dir+\"/tfc_w1_a1_streamlined.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the network has become simplified considerably compared to the previous step -- a lot of nodes have disappeared between the `MatMul` layers, and the `Sign` nodes have been replaced with `MultiThreshold` nodes instead. \n",
    "\n",
    "**The current implementation of streamlining is highly network-specific and may not work for your network if its topology is very different than the example network here. We hope to rectify this in future releases.**\n",
    "\n",
    "Our example network is a quantized network with 1-bit bipolar (-1, +1 values) precision, and we want FINN to implement them as XNOR-popcount operations [as described in the original FINN paper](https://arxiv.org/pdf/1612.07119). For this reason, after streamlining, the resulting bipolar matrix multiplications are converted into xnorpopcount operations. This transformation produces operations that are again collapsed and converted into thresholds. This procedure is shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1a1_ready_for_hls_conversion.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1346f7780>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.transformation.bipolar_to_xnor import ConvertBipolarMatMulToXnorPopcount\n",
    "import finn.transformation.streamline.absorb as absorb\n",
    "from finn.transformation.streamline.round_thresholds import RoundAndClipThresholds\n",
    "\n",
    "model = model.transform(ConvertBipolarMatMulToXnorPopcount())\n",
    "model = model.transform(absorb.AbsorbAddIntoMultiThreshold())\n",
    "model = model.transform(absorb.AbsorbMulIntoMultiThreshold())\n",
    "model = model.transform(RoundAndClipThresholds())\n",
    "\n",
    "model.save(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")\n",
    "showInNetron(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe the pairs of `XnorPopcountmatMul` and `MultiThreshold` layers following each other -- this is the particular pattern that the next step will be looking for in order to convert them to HLS layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conversion to HLS layers <a id='hls_layers'></a>\n",
    "Converts the nodes to HLS layers that correspond to the functions in [finn-hls library](https://finn-hlslib.readthedocs.io/en/latest/). In our case this transformation converts pairs of binary XnorPopcountMatMul layers to StreamingFCLayer_Batch layers. Any immediately following MultiThreshold layers will also be absorbed into the MVTU.\n",
    "\n",
    "Below is the code for the transformation and the network is visualized using netron to create the new structure with `StreamingFCLayer_Batch` nodes, which will correspond to a function call from the [finn-hlslib](https://finn-hlslib.readthedocs.io/en/latest/library/fclayer.html#_CPPv4I_j_j_j_j000_i_i000E22StreamingFCLayer_BatchvRN3hls6streamI7ap_uintI9InStreamWEEERN3hls6streamI7ap_uintI10OutStreamWEEERK2TWRK2TAKjRK1R) library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The transformation `to_hls.InferBinaryStreamingFCLayer` gets the string \"decoupled\" as argument, this indicates the `mem_mode` for the weights. In FINN there are different options to set the way the weights are stored and accessed. For details please have a look on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_hls_layers.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1346f1080>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import finn.transformation.fpgadataflow.convert_to_hls_layers as to_hls\n",
    "model = ModelWrapper(build_dir+\"/tfc_w1a1_ready_for_hls_conversion.onnx\")\n",
    "model = model.transform(to_hls.InferBinaryStreamingFCLayer(\"decoupled\"))\n",
    "model.save(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")\n",
    "showInNetron(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each StreamingFCLayer_Batch node has two attributes that specify the degree of folding, PE and SIMD. In all nodes the values for these attributes are set as default to 1, which would correspond to a maximum folding (time multiplexing) and thus minimum performance. We will shortly cover how these can be adjusted, but first we want to separate the HLS layers from the non-HLS layers in this network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Dataflow Partition <a id='dataflow_partition'></a>\n",
    "\n",
    "In the graph above, you can see that there is a mixture of FINN HLS layers (StreamingFCLayer_Batch) with regular ONNX layers (Reshape, Mul, Add). To create a bitstream, FINN needs a model with only HLS layers. In order to achieve this, we will use the `CreateDataflowPartition` transformation to create a \"dataflow partition\" in this graph, separating out the HLS layers into another model, and replacing them with a placeholder layer called StreamingDataflowPartition:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_dataflow_parent.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1ad0b6e48>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.transformation.fpgadataflow.create_dataflow_partition import CreateDataflowPartition\n",
    "\n",
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1_hls_layers.onnx\")\n",
    "parent_model = model.transform(CreateDataflowPartition())\n",
    "parent_model.save(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")\n",
    "showInNetron(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the StreamingFCLayer instances have all been replaced with a single `StreamingDataflowPartition`, which has an attribute `model` that points to the extracted, HLS dataflow-only graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/tmp/finn_dev_jakobap/dataflow_partition_pbrjefjg/df_model.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1346f3550>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.custom_op.registry import getCustomOp\n",
    "sdp_node = getCustomOp(parent_model.graph.node[2])\n",
    "dataflow_model_filename = sdp_node.get_nodeattr(\"model\")\n",
    "showInNetron(dataflow_model_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see all the extracted `StreamingFCLayer` instances have been moved to the child (dataflow) model. We will load the child model with `ModelWrapper` and continue working on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelWrapper(dataflow_model_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Folding and Datawidth Converter, FIFO and TLastMarker Insertion <a id='folding'></a>\n",
    "\n",
    "*Folding* in FINN describes how much a layer is time-multiplexed in terms of execution resources. There are several *folding factors* for each layer, controlled by the PE (parallelization over outputs) and SIMD (parallelization over inputs) parameters as described by the original [FINN paper](https://arxiv.org/pdf/1612.07119). The higher the PE and SIMD values are set, the faster the generated accelerator will run, and the more FPGA resources it will consume. \n",
    "\n",
    "Since the folding parameters are node attributes, they can be easily accessed and changed using a helper function of the `ModelWrapper`. But first we take a closer look at one of the nodes that implement a StreamingFCLayer_Batch operation. This is where the Netron visualization helps us, in the above diagram we can see that the first four nodes are StreamingFCLayer_Batch. So as an example we extract the first node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the higher-level [HLSCustomOp](https://github.com/Xilinx/finn/blob/master/src/finn/custom_op/fpgadataflow/__init__.py) wrappers for this node. These wrappers provide easy access to specific properties of these nodes, such as the folding factors (PE and SIMD). Let's have a look at which node attributes are defined by the CustomOp wrapper, and adjust the SIMD and PE attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CustomOp wrapper is of class StreamingFCLayer_Batch\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'PE': ('i', True, 0),\n",
       " 'SIMD': ('i', True, 0),\n",
       " 'MW': ('i', True, 0),\n",
       " 'MH': ('i', True, 0),\n",
       " 'resType': ('s', True, ''),\n",
       " 'ActVal': ('i', False, 0),\n",
       " 'inputDataType': ('s', True, ''),\n",
       " 'weightDataType': ('s', True, ''),\n",
       " 'outputDataType': ('s', True, ''),\n",
       " 'binaryXnorMode': ('i', False, 0),\n",
       " 'noActivation': ('i', False, 0),\n",
       " 'numInputVectors': ('ints', False, [1]),\n",
       " 'mem_mode': ('s', False, 'const'),\n",
       " 'ram_style': ('s', False, 'auto'),\n",
       " 'backend': ('s', True, 'fpgadataflow'),\n",
       " 'code_gen_dir_cppsim': ('s', False, ''),\n",
       " 'code_gen_dir_ipgen': ('s', False, ''),\n",
       " 'executable_path': ('s', False, ''),\n",
       " 'ipgen_path': ('s', False, ''),\n",
       " 'ip_path': ('s', False, ''),\n",
       " 'ip_vlnv': ('s', False, ''),\n",
       " 'exec_mode': ('s', False, ''),\n",
       " 'sim_cycles': ('i', False, 0),\n",
       " 'rtlsim_trace': ('s', False, ''),\n",
       " 'res_estimate': ('s', False, ''),\n",
       " 'res_hls': ('s', False, ''),\n",
       " 'res_synth': ('s', False, ''),\n",
       " 'rtlsim_so': ('s', False, ''),\n",
       " 'inFIFODepth': ('i', False, 2),\n",
       " 'outFIFODepth': ('i', False, 2)}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fc0 = model.graph.node[0]\n",
    "fc0w = getCustomOp(fc0)\n",
    "\n",
    "print(\"CustomOp wrapper is of class \" + fc0w.__class__.__name__)\n",
    "\n",
    "fc0w.get_nodeattr_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the PE and SIMD are listed as node attributes, as well as the depths of the FIFOs that will be inserted between consecutive layers, and all can be adjusted using `set_nodeattr` subject to certain constraints.\n",
    "**In this notebook we are setting the folding factors and FIFO depths manually, but in a future version we will support determining the folding factors given an FPGA resource budget according to the analytical model from the [FINN-R paper](https://arxiv.org/pdf/1809.04570).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "fc_layers = model.get_nodes_by_op_type(\"StreamingFCLayer_Batch\")\n",
    "# (PE, SIMD, in_fifo_depth, out_fifo_depth, ramstyle) for each layer\n",
    "config = [\n",
    "    (16, 49, 16, 64, \"block\"),\n",
    "    (8, 8, 64, 64, \"auto\"),\n",
    "    (8, 8, 64, 64, \"auto\"),\n",
    "    (10, 8, 64, 10, \"distributed\"),\n",
    "]\n",
    "for fcl, (pe, simd, ififo, ofifo, ramstyle) in zip(fc_layers, config):\n",
    "    fcl_inst = getCustomOp(fcl)\n",
    "    fcl_inst.set_nodeattr(\"PE\", pe)\n",
    "    fcl_inst.set_nodeattr(\"SIMD\", simd)\n",
    "    fcl_inst.set_nodeattr(\"inFIFODepth\", ififo)\n",
    "    fcl_inst.set_nodeattr(\"outFIFODepth\", ofifo)\n",
    "    fcl_inst.set_nodeattr(\"ram_style\", ramstyle)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are setting PE and SIMD so that each layer has a total folding of 16."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides PE and SIMD three other node attributes are set. `ram_style` specifies how the weights are to be stored (BRAM, LUTRAM, and so on). It can be selected explicitly or with the option `auto` you can let Vivado decide.\n",
    "`inFIFODepth` and `outFIFODepth` specifies the FIFO depths that is needed by the node from the surrounding FIFOs. These attributes are used in the transformation 'InsertFIFO' to insert the appropriate FIFOs between the nodes.\n",
    "\n",
    "But before FIFOs can be added, it must be determined whether datawidth converters (DWC) are required and they must be inserted correctly. Because by setting the folding, the folded output shape of one node may not match the folded input shape of the next node. \n",
    "\n",
    "In the following, first DWCs and then FIFOs are inserted using the corresponding transformations in FINN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.insert_dwc import InsertDWC\n",
    "from finn.transformation.fpgadataflow.insert_fifo import InsertFIFO\n",
    "\n",
    "model = model.transform(InsertDWC())\n",
    "model = model.transform(InsertFIFO())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will run the `InsertTLastMarker` transformation to get a `TLastMarker` node at the output of this graph, which is necessary to run the DMA engines correctly. Using netron we can observe that now the nodes contain the set folding, if necessary a DWC is inserted, inbetween the nodes are FIFOs inserted and the last node is the `TLastMarker` node we insert in the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_set_folding_factors.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe135b84780>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.transformation.fpgadataflow.insert_tlastmarker import InsertTLastMarker\n",
    "model = model.transform(InsertTLastMarker())\n",
    "model.save(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")\n",
    "showInNetron(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This completes the network preparation and the network can be passed on to the next block *Vivado HLS and IPI*, which is described below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Vivado HLS and IPI <a id='vivado'></a>\n",
    "* [Generating HLS Code](#hls_per_layer)\n",
    "* [Synthesizing HLS to IP Blocks](#hls_synth)\n",
    "* [IP Stitching](#ip_stitching)\n",
    "\n",
    "As we will be dealing with FPGA synthesis tools in these tasks, we'll define two helper variables that describe the Xilinx FPGA part name and the PYNQ board name that we are targeting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['Ultra96', 'Pynq-Z1', 'Pynq-Z2', 'ZCU104'])\n"
     ]
    }
   ],
   "source": [
    "# print the names of the supported PYNQ boards\n",
    "from finn.util.basic import pynq_part_map\n",
    "print(pynq_part_map.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this if you have a different PYNQ board, see list above\n",
    "pynq_board = \"Pynq-Z1\"\n",
    "fpga_part = pynq_part_map[pynq_board]\n",
    "target_clk_ns = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating HLS Code <a id='hls_per_layer'></a>\n",
    "This section deals with the generation of an IP block from the different layers. These can then be stitched to a block design that corresponds to the complete model. The single conversion into IP blocks allows a good transparency and we can check the functionality of each IP block and compare it with the behaviour of the corresponding ONNX node. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two transformations are required to generate HLS IP blocks for each layer: \n",
    "* `PrepareIP` which generates the HLS C++ code for the node and a tcl-script which starts the HLS synthesis and exports the design as IP. \n",
    "* `HLSSynthIP` which passes the tcl-script to Vivado HLS and thus performs the actual IP generation. \n",
    "\n",
    "We start off by giving unique node names using the basic transformation `GiveUniqueNodeNames`, and then proceed with the HLS C++ code generation with `PrepareIP`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1_set_folding_factors.onnx\")\n",
    "model = model.transform(GiveUniqueNodeNames())\n",
    "\n",
    "from finn.transformation.fpgadataflow.prepare_ip import PrepareIP\n",
    "model = model.transform(PrepareIP(fpga_part, target_clk_ns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthesizing HLS to IP Blocks <a id='hls_synth'></a>\n",
    "\n",
    "Now that we have generated the HLS code for each layer, we can call the `HLSSynthIP` transformation to convert the generated HLS into Vivado IP blocks. **As this involves calling HLS synthesis, this transformation will run for some time (several minutes).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.hlssynth_ip import HLSSynthIP\n",
    "\n",
    "model = model.transform(HLSSynthIP())\n",
    "model.save(build_dir+\"/tfc_w1_a1_ipgen.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `StreamingFCLayer_Batch` node now has new attributes which can be examined more closely with netron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stopping http://0.0.0.0:8081\n",
      "Serving '/workspace/finn/tfc_w1_a1_ipgen.onnx' at http://0.0.0.0:8081\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"400\"\n",
       "            src=\"http://0.0.0.0:8081/\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7fe1346f7588>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "showInNetron(build_dir+\"/tfc_w1_a1_ipgen.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two additional attributes: \n",
    "* `code_gen_dir_ipgen` which contains the directory path where all the files generated by the ipgen transformations are stored\n",
    "* `ipgen_path` which contains the path to the project directory in which the generated IP block is stored\n",
    "\n",
    "We can further investigate which files are produced by taking a look in this directory. For example for the first StreamingFCLayer_Batch node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StreamingFCLayer_Batch_0_memstream.v  thresh.h\r\n",
      "hls_syn_StreamingFCLayer_Batch_0.tcl  top_StreamingFCLayer_Batch_0.cpp\r\n",
      "ipgen.sh\t\t\t      vivado_hls.log\r\n",
      "memblock_0.dat\t\t\t      weights.npy\r\n",
      "project_StreamingFCLayer_Batch_0\r\n"
     ]
    }
   ],
   "source": [
    "fc0w = getCustomOp(model.graph.node[1])\n",
    "code_gen_dir = fc0w.get_nodeattr(\"code_gen_dir_ipgen\")\n",
    "!ls {code_gen_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Directory *project_StreamingFCLayer_Batch_0* contains the project created by Vivado HLS into which the IP Block is exported, along with other files generated by Vivado HLS. If we compare it to the above visualization of the network with netron, this is exactly the name of the folder stored in the node attribute `ipgen_path`. The .cpp code that is passed to Vivado HLS can be found in the file *top_StreamingFCLayer_Batch_0.cpp*. The file *thresh.h* belongs to that as well, it contains the value for the thresholds. The weights are stored as .npy file and as .dat file (*memblock_0.dat*). *vivado_hls.log* is the log file from Vivado HLS. Besides these files, the folder contains *ipgen.sh* and *hls_syn_StreamingFCLayer_Batch_0.tcl* and because we use the StreamingFCLayer in \"decoupled\" mode a verilog wrapper (*StreamingFCLayer_Batch_0_memstream.v*) is produced, for more details on \"decoupled\" and \"const\" mode please see on the [FINN readthedocs website](https://finn.readthedocs.io/) under Internals. \n",
    "\n",
    "In the following we take a closer look at the two generated scripts. We start with *ipgen.sh*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/bin/bash \r\n",
      "cd /tmp/finn_dev_jakobap/code_gen_ipgen_StreamingFCLayer_Batch_0_edb__5oc\r\n",
      "vivado_hls /tmp/finn_dev_jakobap/code_gen_ipgen_StreamingFCLayer_Batch_0_edb__5oc/hls_syn_StreamingFCLayer_Batch_0.tcl\r\n",
      "cd /workspace/finn\r\n"
     ]
    }
   ],
   "source": [
    "shell_script = code_gen_dir + \"/ipgen.sh\"\n",
    "!cat {shell_script}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The script consists only of two framing `cd` commands and a command to pass the tcl script to *vivado_hls*. The directory has to be changed to create the files in the correct folder and will then be changed back to the original directory. \n",
    "\n",
    "Below is the tcl script which is passed to *vivado_hls*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "set config_proj_name project_StreamingFCLayer_Batch_0\r\n",
      "puts \"HLS project: $config_proj_name\"\r\n",
      "set config_hwsrcdir \"/tmp/finn_dev_jakobap/code_gen_ipgen_StreamingFCLayer_Batch_0_edb__5oc\"\r\n",
      "puts \"HW source dir: $config_hwsrcdir\"\r\n",
      "set config_proj_part \"xc7z020clg400-1\"\r\n",
      "\r\n",
      "set config_bnnlibdir \"/workspace/finn-hlslib\"\r\n",
      "\r\n",
      "set config_toplevelfxn \"StreamingFCLayer_Batch_0\"\r\n",
      "set config_clkperiod 10\r\n",
      "\r\n",
      "open_project $config_proj_name\r\n",
      "add_files $config_hwsrcdir/top_StreamingFCLayer_Batch_0.cpp -cflags \"-std=c++0x -I$config_bnnlibdir\"\r\n",
      "\r\n",
      "set_top $config_toplevelfxn\r\n",
      "open_solution sol1\r\n",
      "set_part $config_proj_part\r\n",
      "\r\n",
      "config_interface -m_axi_addr64\r\n",
      "config_rtl -auto_prefix\r\n",
      "\r\n",
      "\r\n",
      "create_clock -period $config_clkperiod -name default\r\n",
      "csynth_design\r\n",
      "export_design -format ip_catalog\r\n",
      "exit 0\r\n"
     ]
    }
   ],
   "source": [
    "tcl_script = code_gen_dir + \"/hls_syn_StreamingFCLayer_Batch_0.tcl\"\n",
    "!cat {tcl_script}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the first part of the script the project is configured. For example the FPGA part and the clock are set. Then the project is opened and the files are added. The toplevel function is set and after creating a clock, the design is first synthesized with `csynth` and then exported as an IP block.\n",
    "\n",
    "Now that all IP blocks are in place, they can be stitched together to create an IP design that matches the ONNX model. This is covered in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IP Stitching <a id='ip_stitching'></a>\n",
    "\n",
    "We now have IP blocks for each of our layers, and will stitch them together into a larger IP that implements the whole network using the `CreateStitchedIP` transformation. Bear in mind that this transformation can only be applied on a graph that only contains HLS nodes that already have been through the `HLSSynthIP` transformation, which is the last step we performed. Prior to calling IP stitching, we'll also use the `ReplaceVerilogRelPaths` transformation to convert any relative `$readmemh` paths in the generated IP blocks to absolute ones, which prevents errors later on. **This step invokes Vivado and may take a few minutes to run.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.create_stitched_ip import CreateStitchedIP\n",
    "from finn.transformation.fpgadataflow.replace_verilog_relpaths import ReplaceVerilogRelPaths\n",
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1_ipgen.onnx\")\n",
    "model = model.transform(ReplaceVerilogRelPaths())\n",
    "model = model.transform(CreateStitchedIP(fpga_part))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you examine the nodes themselves on the transformed model you won't see a difference, because the IP stitching adds model-level metadata to the graph. This can be accessed using the `.model.metadata_props`, the `get_metadata_prop` function in `ModelWrapper`, or by clicking on the global input/output tensors in Netron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[key: \"vivado_stitch_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl\"\n",
       ", key: \"vivado_stitch_vlnv\"\n",
       "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
       ", key: \"wrapper_filename\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
       "]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.model.metadata_props"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_metadata_prop(\"vivado_stitch_proj\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you navigate to the folder above (remember the /tmp/finn_xxx folder is mounted on the host as well as inside Docker) you can open the Vivado project (.xpr) file there using Vivado, and view the following stitched IP block design:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](stitched_ip.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(build_dir+\"/tfc_w1_a1_ipstitch.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, one could take the generated stitched IP and integrate it into your own project using Vivado IP Integrator if desired. Here, we will continue the tutorial by assuming that we want to do a stand-alone deployment for this accelerator for a PYNQ board."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.  PYNQ hardware generation and deployment <a id='hw_test'></a>\n",
    "\n",
    "* [Inserting the IP into a PYNQ Overlay Shell](#pynq_shell)\n",
    "* [Synthesis, Place and Route](#synth_pl_ro)\n",
    "* [Driver Generation](#driver_gen)\n",
    "* [Deployment and Remote Execution](#deploy)\n",
    "* [Throughput Test on PYNQ Board](#throughput)\n",
    "\n",
    "\n",
    "We are almost done preparing our hardware design. We'll now put it in a form suitable for use as a PYNQ overlay, synthesize and deploy it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inserting the IP into a PYNQ Overlay Shell <a id='pynq_shell'></a>\n",
    "\n",
    "To deploy our accelerator on a PYNQ platform, it needs to be put inside an appropriate *shell* that bridges it with the interfaces that the underlying system exposes. FINN makes it easy to create a PYNQ-compatible overlay by inserting the stitched IP into an appropriate PYNQ shell with the `MakePYNQProject` transformation, and view the created PYNQ shell project directory using the `metadata_props`. **This invokes Vivado and may take a few minutes to run.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[key: \"vivado_stitch_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl\"\n",
       ", key: \"vivado_stitch_vlnv\"\n",
       "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
       ", key: \"wrapper_filename\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
       ", key: \"vivado_pynq_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs\"\n",
       ", key: \"vivado_synth_rpt\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs/synth_report.xml\"\n",
       "]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.transformation.fpgadataflow.make_pynq_proj import MakePYNQProject\n",
    "model = ModelWrapper(build_dir+\"/tfc_w1_a1_ipstitch.onnx\")\n",
    "model = model.transform(MakePYNQProject(pynq_board))\n",
    "model.model.metadata_props"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ip_config.tcl\t resizer.cache\tresizer.ip_user_files  resizer.xpr\r\n",
      "make_project.sh  resizer.hw\tresizer.srcs\t       synth_project.sh\r\n"
     ]
    }
   ],
   "source": [
    "! ls {model.get_metadata_prop(\"vivado_pynq_proj\")}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we open the created Vivado project (.xpr) under the `vivado_pynq_proj` directory above, we can see the system-level block design as below, with the FINN-generated part of the design highlighted. Various other components, such as the DMA engine and data width converters, have also been instantiated.\n",
    "![](pynq_shell_project.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(build_dir + \"/tfc_w1_a1_pynq_project.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthesis, Place and Route <a id='synth_pl_ro'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready for the final hardware generation step, which is synthesis, place and route to generate an FPGA bitfile. This can be done by either running the `synth_project.sh` script in the generated Vivado PYNQ project directory inside Docker, or by executing the `SynthPYNQProject` transformation. **This step involves launching Vivado for synthesis and may take a few hours.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[key: \"vivado_stitch_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl\"\n",
       ", key: \"vivado_stitch_vlnv\"\n",
       "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
       ", key: \"wrapper_filename\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
       ", key: \"vivado_pynq_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs\"\n",
       ", key: \"vivado_synth_rpt\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs/synth_report.xml\"\n",
       ", key: \"vivado_pynq_bitfile\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs/resizer.bit\"\n",
       "]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from finn.transformation.fpgadataflow.synth_pynq_proj import SynthPYNQProject\n",
    "model = ModelWrapper(build_dir + \"/tfc_w1_a1_pynq_project.onnx\")\n",
    "model = model.transform(SynthPYNQProject())\n",
    "model.model.metadata_props"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(build_dir + \"/tfc_w1_a1_post_synthesis.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Driver Generation <a id='driver_gen'></a>\n",
    "\n",
    "Now that we have synthesized a bitfile for our network, we will generate some Python code for PYNQ that will act as the driver for this bitfile, package everything into a deployment folder and copy that to our PYNQ board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.make_pynq_driver import MakePYNQDriver\n",
    "model = ModelWrapper(build_dir + \"/tfc_w1_a1_post_synthesis.onnx\")\n",
    "model = model.transform(MakePYNQDriver())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated driver is placed in a folder that is indicated by the `pynq_driver_dir` top-level metadata. We can examine the generated PYNQ Python driver code as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r\n",
      "import argparse\r\n",
      "\r\n",
      "from pynq import Overlay\r\n",
      "import numpy as np\r\n",
      "from pynq import allocate\r\n",
      "import time\r\n",
      "from finn.util.data_packing import (\r\n",
      "    finnpy_to_packed_bytearray,\r\n",
      "    packed_bytearray_to_finnpy\r\n",
      ")\r\n",
      "from finn.core.datatype import DataType\r\n",
      "\r\n",
      "class FINNAccelDriver():\r\n",
      "    def __init__(self, N, bitfile):\r\n",
      "        \"\"\"Instantiate the FINN accelerator driver.\r\n",
      "        Gets batchsize (N) as integer and path to bitfile as string.\"\"\"\r\n",
      "        self.N = N\r\n",
      "        # input FINN DataType\r\n",
      "        self.idt = DataType.BINARY\r\n",
      "        # output FINN DataType\r\n",
      "        self.odt = DataType.UINT32\r\n",
      "        # input and output shapes\r\n",
      "        self.ishape_normal = (N, 784)\r\n",
      "        self.oshape_normal = (N, 10)\r\n",
      "        self.ishape_folded = (N, 16, 49)\r\n",
      "        self.oshape_folded = (N, 1, 10)\r\n",
      "        self.ishape_packed = (N, 16, 7)   # datatype np.uint8\r\n",
      "        self.oshape_packed = (N, 1, 40)  # datatype np.uint8\r\n",
      "        # load bitfile and set up accelerator\r\n",
      "        self.ol = Overlay(bitfile)\r\n",
      "        self.dma = self.ol.axi_dma_0\r\n",
      "        self.ctrl_regs = self.ol.resize_accel_0\r\n",
      "        # neuron folding factor of output = iterations per sample\r\n",
      "        self.itersPerSample = self.oshape_packed[-2]\r\n",
      "        # AXI lite register offset for number of iterations\r\n",
      "        # used by TLastMarker to signal end of transmission for AXI CDMA\r\n",
      "        self.REG_OFFSET_NUM_ITERS = 0x10\r\n",
      "        # set up TLastMarker with correct num. samples\r\n",
      "        self.ctrl_regs.write(self.REG_OFFSET_NUM_ITERS, self.N*self.itersPerSample)\r\n",
      "\r\n",
      "        # allocate a PYNQ buffer for the packed input and buffer\r\n",
      "        self.ibuf_packed_device = allocate(shape=self.ishape_packed, dtype=np.uint8)\r\n",
      "        self.obuf_packed_device = allocate(shape=self.oshape_packed, dtype=np.uint8)\r\n",
      "\r\n",
      "    def fold_input(self, ibuf_normal):\r\n",
      "        \"\"\"Reshapes input in desired shape.\r\n",
      "        Gets input data (ibuf_normal), checks if data is in expected normal shape.\r\n",
      "        Returns folded input.\"\"\"\r\n",
      "        # ensure that shape is as expected\r\n",
      "        assert ibuf_normal.shape == self.ishape_normal\r\n",
      "        # convert to folded form\r\n",
      "        ibuf_folded = ibuf_normal.reshape(self.ishape_folded)\r\n",
      "        return ibuf_folded\r\n",
      "\r\n",
      "    def pack_input(self, ibuf_folded):\r\n",
      "        \"\"\"Packs folded input and reverses both SIMD dim and endianness.\r\n",
      "        Gets input data in folded shape and returns packed input data.\"\"\"\r\n",
      "        ibuf_packed = finnpy_to_packed_bytearray(\r\n",
      "            ibuf_folded, self.idt, reverse_endian=True, reverse_inner=True\r\n",
      "        )\r\n",
      "        return ibuf_packed\r\n",
      "\r\n",
      "    def unpack_output(self, obuf_packed):\r\n",
      "        \"\"\"Unpacks the packed output buffer from accelerator.\r\n",
      "        Gets packed output and returns output data in folded shape.\"\"\"\r\n",
      "        obuf_folded = packed_bytearray_to_finnpy(\r\n",
      "            obuf_packed, self.odt, self.oshape_folded, reverse_endian=True, reverse_inner=True\r\n",
      "        )\r\n",
      "        return obuf_folded\r\n",
      "\r\n",
      "    def unfold_output(self, obuf_folded):\r\n",
      "        \"\"\"Unfolds output data to normal shape.\r\n",
      "        Gets folded output data and returns output data in normal shape.\"\"\"\r\n",
      "        obuf_normal = obuf_folded.reshape(self.oshape_normal)\r\n",
      "        return obuf_normal\r\n",
      "\r\n",
      "    def copy_input_data_to_device(self, data):\r\n",
      "        \"\"\"Copies given input data to PYNQ buffer.\"\"\"\r\n",
      "        np.copyto(self.ibuf_packed_device, data)\r\n",
      "\r\n",
      "    def execute(self):\r\n",
      "        \"\"\"Executes accelerator by setting up the DMA and\r\n",
      "        waiting until all transfers complete. Uses only member variables and\r\n",
      "        returns nothing.\"\"\"\r\n",
      "        dma = self.dma\r\n",
      "        dma.sendchannel.transfer(self.ibuf_packed_device)\r\n",
      "        dma.recvchannel.transfer(self.obuf_packed_device)\r\n",
      "        dma.sendchannel.wait()\r\n",
      "        dma.recvchannel.wait()\r\n",
      "\r\n",
      "\r\n",
      "if __name__ == \"__main__\":\r\n",
      "    parser = argparse.ArgumentParser(description='Set exec mode, batchsize N, bitfile name, inputfile name and outputfile name')\r\n",
      "    parser.add_argument('--exec_mode', help='Please select functional verification (\"execute\") or throughput test (\"throughput_test\")', default=\"execute\")\r\n",
      "    parser.add_argument('--batchsize', help='number of samples for inference', type=int, default=1)\r\n",
      "    parser.add_argument('--bitfile', help='name of bitfile (i.e. \"resizer.bit\")', default=\"resizer.bit\")\r\n",
      "    parser.add_argument('--inputfile', help='name of input npy file (i.e. \"input.npy\")', default=\"input.npy\")\r\n",
      "    parser.add_argument('--outputfile', help='name of output npy file (i.e. \"output.npy\")', default=\"output.npy\")\r\n",
      "    # parse arguments\r\n",
      "    args = parser.parse_args()\r\n",
      "    exec_mode = args.exec_mode\r\n",
      "    N = args.batchsize\r\n",
      "    bitfile = args.bitfile\r\n",
      "    inputfile = args.inputfile\r\n",
      "    outputfile = args.outputfile\r\n",
      "\r\n",
      "    # instantiate FINN accelerator driver and pass batchsize and bitfile\r\n",
      "    finnDriver = FINNAccelDriver(N, bitfile)\r\n",
      "\r\n",
      "    # for the remote execution the data from the input npy file has to be loaded,\r\n",
      "    # packed and copied to the PYNQ buffer\r\n",
      "    if exec_mode == \"execute\":\r\n",
      "        # load desired input .npy file\r\n",
      "        ibuf_normal = np.load(inputfile)\r\n",
      "        ibuf_folded = finnDriver.fold_input(ibuf_normal)\r\n",
      "        ibuf_packed = finnDriver.pack_input(ibuf_folded)\r\n",
      "        finnDriver.copy_input_data_to_device(ibuf_packed)\r\n",
      "    elif exec_mode != \"throughput_test\":\r\n",
      "        raise Exception(\"Exec mode has to be set to remote_pynq or throughput_test\")\r\n",
      "\r\n",
      "    # for the throughput test the runtime of the network has to be measured\r\n",
      "    if exec_mode == \"throughput_test\":\r\n",
      "        # measure runtime of network\r\n",
      "        start = time.time()\r\n",
      "        # dictionary for results of throughput test\r\n",
      "        res={}\r\n",
      "\r\n",
      "    # execute accelerator\r\n",
      "    finnDriver.execute()\r\n",
      "\r\n",
      "    # measure run time and fill dictionary with results of the throughput test\r\n",
      "    if exec_mode == \"throughput_test\":\r\n",
      "        end = time.time()\r\n",
      "        runtime = end - start\r\n",
      "        res[\"runtime[ms]\"] = runtime*1000\r\n",
      "        res[\"throughput[images/s]\"] = N / runtime\r\n",
      "        res[\"DRAM_in_bandwidth[Mb/s]\"] = np.prod(finnDriver.ishape_packed)*0.000001 / runtime\r\n",
      "        res[\"DRAM_out_bandwidth[Mb/s]\"] = np.prod(finnDriver.oshape_packed)*0.000001 / runtime\r\n",
      "        file = open(\"nw_metrics.txt\", \"w\")\r\n",
      "        file.write(str(res))\r\n",
      "        file.close()\r\n",
      "\r\n",
      "    # if execution is selected unpack, unfold and save output to output npy file\r\n",
      "    else:\r\n",
      "        obuf_folded = finnDriver.unpack_output(finnDriver.obuf_packed_device)\r\n",
      "        obuf_normal = finnDriver.unfold_output(obuf_folded)\r\n",
      "        np.save(outputfile, obuf_normal)\r\n",
      "\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "driver_dir = model.get_metadata_prop(\"pynq_driver_dir\")\n",
    "! cat {driver_dir}/driver.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that in the generated driver a class is implemented which implements the FINN accelerator. The constructor gets the batchsize (N) as integer and the bitfile as string. It also contains the expected input/output shapes, and takes care of the instantiation of the accelerator by loading the bitfile and setting up dma and buffer. Several member functions take care of the data folding and packing. The function `copy_input_data_to_device` copies the input data into the PYNQ buffer and `execute` sets up the dma channels and waits until the transfer is completed. This class is used in the main function. But first the arguments are parsed, which are passed to the script. The driver can be used in two modes: \"execute\" and \"throughput_test\". By default all arguments are set to \"execute\" mode. In this mode the batch size is 1, and the passed files are set to the names used by the FINN transformations.\n",
    "\n",
    "In the \"execute\" mode works as follows:\n",
    "1. the data is loaded from the \"inputfile\"\n",
    "2. the data is folded using `fold_input`\n",
    "3. the data is packed using `pack_input`\n",
    "4. the data is copied to the device using `copy_input_data_to_device`\n",
    "5. FINNAccelDriver is executed using `execute`\n",
    "6. the data is unpacked using `unpack_output`\n",
    "7. the data is unfolded using `unfold_output`\n",
    "8. the data is stored in the \"outputfile\"\n",
    "\n",
    "If \"throughput_test\" is selected as `exec_mode`, no actual data needs to be loaded. The batchsize N should be set to a high value (i.e. 1000) and a time measurement is implemented in python. An empty dictionary (`res`) is created and after running the accelerator with the measured runtime it is filled with the metrics and saved in a .txt file.\n",
    "\n",
    "You can build your own applications around the accelerator by modifying the driver, or use the remote execution capabilities that FINN provides just to check if it is working, which will be our next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deployment and Remote Execution <a id='deploy'></a>\n",
    "\n",
    "We'll now use the `DeployToPYNQ` transformation to create a deployment folder with the bitfile and driver file(s), and copy that to the PYNQ board. You can change the default IP address, username, password and target folder for the PYNQ below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from finn.transformation.fpgadataflow.make_deployment import DeployToPYNQ\n",
    "ip = \"192.168.3.1\"\n",
    "port = \"22\"\n",
    "username = \"xilinx\"\n",
    "password = \"xilinx\"\n",
    "target_dir = \"/home/xilinx/finn_tfc_end2end_example\"\n",
    "model = model.transform(DeployToPYNQ(ip, port, username, password, target_dir))\n",
    "model.save(build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify that the remote access credentials is saved in the model metadata, and that the deployment folder has been successfully copied to the board:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[key: \"vivado_stitch_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl\"\n",
       ", key: \"vivado_stitch_vlnv\"\n",
       "value: \"xilinx_finn:finn:finn_design:1.0\"\n",
       ", key: \"wrapper_filename\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_stitch_proj_oa43bqzl/finn_vivado_stitch_proj.srcs/sources_1/bd/finn_design/hdl/finn_design_wrapper.v\"\n",
       ", key: \"vivado_pynq_proj\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs\"\n",
       ", key: \"vivado_synth_rpt\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs/synth_report.xml\"\n",
       ", key: \"vivado_pynq_bitfile\"\n",
       "value: \"/tmp/finn_dev_jakobap/vivado_pynq_proj_ljn53hfs/resizer.bit\"\n",
       ", key: \"pynq_driver_dir\"\n",
       "value: \"/tmp/finn_dev_jakobap/pynq_driver_j_9suyqm\"\n",
       ", key: \"pynq_ip\"\n",
       "value: \"51.37.47.42\"\n",
       ", key: \"pynq_port\"\n",
       "value: \"23\"\n",
       ", key: \"pynq_username\"\n",
       "value: \"xilinx\"\n",
       ", key: \"pynq_password\"\n",
       "value: \"x1l1nx_f!nn\"\n",
       ", key: \"pynq_target_dir\"\n",
       "value: \"/home/xilinx/finn_tfc_end2end_example\"\n",
       ", key: \"pynq_deployment_dir\"\n",
       "value: \"/tmp/finn_dev_jakobap/pynq_deployment_962qxwkv\"\n",
       ", key: \"pynq_deploy_dir\"\n",
       "value: \"/tmp/finn_dev_jakobap/pynq_deployment_962qxwkv\"\n",
       ", key: \"exec_mode\"\n",
       "value: \"remote_pynq\"\n",
       "]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.model.metadata_props"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/xilinx/finn_tfc_end2end_example/pynq_deployment_26e8h5jo:\r\n",
      "total 4276\r\n",
      "-rw-r--r-- 1 xilinx xilinx    6363 May  7 10:35 driver.py\r\n",
      "drwxr-xr-x 4 xilinx xilinx    4096 May  7 10:35 finn\r\n",
      "-rw-r--r-- 1 xilinx xilinx    3264 May  7 10:55 input.npy\r\n",
      "-rw-r--r-- 1 root   root       172 May  7 10:37 nw_metrics.txt\r\n",
      "-rw-r--r-- 1 root   root       120 May  7 10:55 output.npy\r\n",
      "-rw-r--r-- 1 xilinx xilinx 4045675 May  7 10:35 resizer.bit\r\n",
      "-rw-r--r-- 1 xilinx xilinx  302015 May  7 10:35 resizer.hwh\r\n",
      "-rw-r--r-- 1 root   root        32 May  7 10:55 sds_trace_data.dat\r\n",
      "\r\n",
      "/home/xilinx/finn_tfc_end2end_example/pynq_deployment_962qxwkv:\r\n",
      "total 4260\r\n",
      "-rw-r--r-- 1 xilinx xilinx    6363 May  7 17:44 driver.py\r\n",
      "drwxr-xr-x 4 xilinx xilinx    4096 May  7 17:44 finn\r\n",
      "-rw-r--r-- 1 xilinx xilinx 4045675 May  7 17:44 resizer.bit\r\n",
      "-rw-r--r-- 1 xilinx xilinx  302015 May  7 17:44 resizer.hwh\r\n",
      "\r\n",
      "/home/xilinx/finn_tfc_end2end_example/pynq_deployment_kvurnk0c:\r\n",
      "total 4300\r\n",
      "-rw-r--r-- 1 xilinx xilinx    3861 Apr 27 12:36 driver.py\r\n",
      "drwxr-xr-x 4 xilinx xilinx    4096 Apr 27 12:37 finn\r\n",
      "-rw-r--r-- 1 xilinx xilinx    3264 Apr 27 12:37 input.npy\r\n",
      "-rw-r--r-- 1 root   root        78 Apr 27 12:38 nw_metrics.txt\r\n",
      "-rw-r--r-- 1 root   root       120 Apr 27 12:37 output.npy\r\n",
      "-rw-r--r-- 1 xilinx xilinx 4045675 Apr 27 12:36 resizer.bit\r\n",
      "-rw-r--r-- 1 xilinx xilinx  329531 Apr 27 12:36 resizer.hwh\r\n",
      "-rw-r--r-- 1 root   root        32 Apr 27 12:38 sds_trace_data.dat\r\n",
      "\r\n",
      "/home/xilinx/finn_tfc_end2end_example/pynq_deployment__tnbutz_:\r\n",
      "total 4276\r\n",
      "-rw-r--r-- 1 xilinx xilinx    6363 May  6 17:34 driver.py\r\n",
      "drwxr-xr-x 4 xilinx xilinx    4096 May  6 17:34 finn\r\n",
      "-rw-r--r-- 1 xilinx xilinx    3264 May  6 17:34 input.npy\r\n",
      "-rw-r--r-- 1 root   root       173 May  6 17:35 nw_metrics.txt\r\n",
      "-rw-r--r-- 1 root   root       120 May  6 17:34 output.npy\r\n",
      "-rw-r--r-- 1 xilinx xilinx 4045675 May  6 17:34 resizer.bit\r\n",
      "-rw-r--r-- 1 xilinx xilinx  302015 May  6 17:34 resizer.hwh\r\n",
      "-rw-r--r-- 1 root   root        32 May  6 17:35 sds_trace_data.dat\r\n",
      "\r\n",
      "/home/xilinx/finn_tfc_end2end_example/pynq_deployment_w4aa1r9k:\r\n",
      "total 4276\r\n",
      "-rw-r--r-- 1 xilinx xilinx    6363 May  7 15:05 driver.py\r\n",
      "drwxr-xr-x 4 xilinx xilinx    4096 May  7 15:05 finn\r\n",
      "-rw-r--r-- 1 xilinx xilinx    3264 May  7 15:06 input.npy\r\n",
      "-rw-r--r-- 1 root   root       172 May  7 15:11 nw_metrics.txt\r\n",
      "-rw-r--r-- 1 root   root       120 May  7 15:06 output.npy\r\n",
      "-rw-r--r-- 1 xilinx xilinx 4045675 May  7 15:05 resizer.bit\r\n",
      "-rw-r--r-- 1 xilinx xilinx  302015 May  7 15:05 resizer.hwh\r\n",
      "-rw-r--r-- 1 root   root        32 May  7 15:11 sds_trace_data.dat\r\n"
     ]
    }
   ],
   "source": [
    "! sshpass -p {password} ssh {username}@{ip} -p {port} 'ls -l {target_dir}/*'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only have two more steps to be able to remotely execute the deployed bitfile with some test data from the MNIST dataset. Let's load up some test data that comes bundled with FINN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fe11dda48d0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAARX0lEQVR4nO3dfYyVZXrH8d/FoDAw8iYRCaisG/5QqmUbgk1KyOKmxlUMbKJm/aPauAmarMmqTVqz/UOSaqJVa/pH3YStL9CsmiWoq0a7a82mWo1GNFQQW1CULGR4E5H3t+HqH/NgZ3We6549z3nOc9z7+0kmM3Ouec65OTM/zsv13Pdt7i4Af/xGNT0AAJ1B2IFMEHYgE4QdyARhBzIxupM3Zma89Z+ZUaPKH09OnTpV23VXvf6enp6wPjAw0PJ1183dbbjLK4XdzK6U9M+SeiT9q7vfV+X6cmU27O/mS6k/6ip/eKNHx38CqcCk6r29vaW1Q4cOhcem9PX1hfUDBw6U1lIt50mTJoX1zz77LKx3o5afxptZj6R/kfR9SRdLusHMLm7XwAC0V5XX7PMlfeTuW9z9uKSnJS1pz7AAtFuVsM+Q9Lsh328rLvs9ZrbMzNaa2doKtwWgotrfoHP3FZJWSLxBBzSpyiP7dknnDfl+ZnEZgC5UJezvSJptZt8yszMl/VDS8+0ZFoB2a/lpvLufNLPbJP1ag623x9z9g7aNLCPjx48P6wcPHmz5useMGRPWjx07FtZTbcFx48aF9ai9lmoppqSOj9prqT76vn37WhpTN6v0mt3dX5L0UpvGAqBGnC4LZIKwA5kg7EAmCDuQCcIOZIKwA5mwTq4um+vpsqled6qXffTo0bA+duzYlo9Nia676vWfffbZYb3qNNLofp06dWp47O7du8N6amrwyZMnw3qdyuaz88gOZIKwA5kg7EAmCDuQCcIOZIKwA5mg9fYNkGrNVfkd1nnddUtNDa6yem1q6m5qanCTS03TegMyR9iBTBB2IBOEHcgEYQcyQdiBTBB2IBP02TvgrLPOCuvRbqOSNHHixLB+4sSJ0lpqN9LUFNbPP/88rC9YsCCs33rrraW1VC/6jjvuCOtbt24N601OM20SfXYgc4QdyARhBzJB2IFMEHYgE4QdyARhBzJBn/0b4JFHHgnrUS871Wuuuox1b29vWI+ktk2+5JJLwvqmTZvC+vHjx0trZ5xxRnhsdO6ClP53HzlyJKzXqazPXmnLZjP7VNIBSQOSTrr7vCrXB6A+lcJeWOTue9pwPQBqxGt2IBNVw+6SfmNm75rZsuF+wMyWmdlaM1tb8bYAVFD1afwCd99uZudIesXM/sfdXxv6A+6+QtIKiTfogCZVemR39+3F512SnpU0vx2DAtB+LYfdzMab2Vmnv5Z0haQN7RoYgPaq8jR+mqRniz7taElPuvu/t2VUf2RSWzYvWrQorF922WVhPeqVHzx4MDw21W/u6+sL66nzNKI566m11x999NGWr1uS7rzzztLaW2+9FR5b93bSTWg57O6+RdKftnEsAGpE6w3IBGEHMkHYgUwQdiAThB3IBFNcu0Bqqubs2bPD+v79+0trEyZMCI+NpoFK6SmwVbZ8TrX9UlJLcO/du7e0tnTp0vDYdevWhfVUSzLV8qwTS0kDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJdiw42TFRT7fOfnBK6thU/ZZbbgnrq1atCuszZ85s+bZTffZ77rknrK9evTqsn3nmmaW1K664Ijz2wQcfDOuprbCj2168eHF47LZt28L6nj3fvDVWeWQHMkHYgUwQdiAThB3IBGEHMkHYgUwQdiATHZ/Pnup3Rzo51naqOvd54cKFYf2iiy4qrY0bNy48dvTo+FSLNWvWhPUtW7aE9SpSyz3PmTMnrKfu90jq75T57AC6FmEHMkHYgUwQdiAThB3IBGEHMkHYgUx0vM8+alT5/y9V54XXqcpc+lOnTlW67eg+S9VPnjwZHjt+/PiwfujQobCe2o46+p2l5tJfffXVYf3pp58O61X67Kk17VP3a5Na7rOb2WNmtsvMNgy5bIqZvWJmm4vPk9s5WADtN5Kn8U9IuvIrl90l6VV3ny3p1eJ7AF0sGXZ3f03SV/fRWSJpZfH1SknxXjoAGtfqGnTT3L2/+HqHpGllP2hmyyQta/F2ALRJ5QUn3d2jDRvdfYWkFRIbOwJNarX1ttPMpktS8XlX+4YEoA6thv15STcVX98k6VftGQ6AuiT77Gb2lKTvSpoqaaekuyU9J+mXks6XtFXS9e5evhn2/19XbU/jq64bX7UeSfVkU3uoR/uvV9Xb2xvWjxw5EtZT5wBUOcfgwgsvDOsff/xxy9edGldqTfqUw4cPVzq+irI+e/I1u7vfUFL6XqURAegoTpcFMkHYgUwQdiAThB3IBGEHMsGWzYVUC3JgYCCsR3p6esJ61WWHozZRqsWUmsKakrr+aNvkqCZJixYtamlMp0W/0xMnToTHpqa4Vvl7aAqP7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZKKr+ux1budcdTnnKuq+7QMHDpTWUv3iVK87dXyqTx8tF51axvq6664L60ePHg3rY8eOLa2l+uyp31mTWzK3ikd2IBOEHcgEYQcyQdiBTBB2IBOEHcgEYQcy0fE+ezS3u5t75dGSyanllFPq3Fb50ksvDY+dM2dOWE8tJf3cc8+F9UjUB5ekhQsXhvUqW3inlqGOzl2Qqi/B3QQe2YFMEHYgE4QdyARhBzJB2IFMEHYgE4QdyETH++zRnPU6++ipufKped1RT3j06PhuXLp0aVhPHb9kyZKwPmbMmNLa3Llzw2MnTZoU1lO97Ndff73l42fPnh0em1qbPdXrXr9+fWnt8ssvD4+N7lOpO/voKclHdjN7zMx2mdmGIZctN7PtZrau+Liq3mECqGokT+OfkHTlMJc/7O5zi4+X2jssAO2WDLu7vyZpbwfGAqBGVd6gu83M3i+e5k8u+yEzW2Zma81sbYXbAlBRq2H/maRvS5orqV/SQ2U/6O4r3H2eu89r8bYAtEFLYXf3ne4+4O6nJP1c0vz2DgtAu7UUdjObPuTbH0jaUPazALqDpfqoZvaUpO9Kmippp6S7i+/nSnJJn0q6xd37kzdmFt5Yqt+cmvcdmTVrVli/5pprwvrixYtLa6l516l526m509H+61K8hnlfX194bErVed3R7/SLL74Ij504cWJYT9m8eXNpbdWqVeGxDz1U+spUUnf32d192JNKkifVuPsNw1z8aOURAegoTpcFMkHYgUwQdiAThB3IBGEHMpFsvbX1xsw8Wna5zimud999d1hfvnx5WN+zZ09pberUqa0M6UuprYf37o2nJkT1Cy64IDw21RZMbdmccuzYsdJaahpp6u8h1YqNpi2ntlx++eWXw/rNN98c1pvc0rms9cYjO5AJwg5kgrADmSDsQCYIO5AJwg5kgrADmeh4nz2qV9maODXVMtX3rLLt8q5du8L61q1bw/oDDzwQ1levXh3W580rXwTo4YcfDo9Nbdk8eXLpimOSpG3btoX16Hf6xBNPhMd+8sknYf3aa68N69HU46rTa1988cWwnpoyXSf67EDmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZKKjffZRo0Z5ND/6+PHj4fHnnHNOaW337t3hsak+e2rudNQvTm0HvWnTprA+ZcqUsJ5atjha7vn8888Pj03NZ08t771v376wfuONN5bWXnjhhfDYlNQ6AtFy0YsWLQqPTa0xkLpfUst/14k+O5A5wg5kgrADmSDsQCYIO5AJwg5kgrADmeiq+exVpPqeK1euDOvXX399y9d/+PDh8Nhx48aF9dS2yKl5/gMDA6W11Lrvb775Zlh/8sknw/q6devC+htvvFFaS51fkOrhp37n0Xkb8+fPD499++23w/rjjz8e1lPrytep5T67mZ1nZr81s41m9oGZ/aS4fIqZvWJmm4vP8SoHABo1kqfxJyX9jbtfLOnPJf3YzC6WdJekV919tqRXi+8BdKlk2N29393fK74+IOlDSTMkLZF0+rnxSklL6xokgOriFz1fYWazJH1H0tuSprl7f1HaIWlayTHLJC1rfYgA2mHE78abWZ+kNZJud/f9Q2s++C7fsG++ufsKd5/n7uWrIgKo3YjCbmZnaDDov3D3Z4qLd5rZ9KI+XVK8xCqARiVbbzY4f3OlpL3ufvuQyx+Q9Jm732dmd0ma4u5/m7iu8MbOPffccCw7duwI65Fo+15JmjlzZli/9957S2szZswIj01tuZzaujjaLlqS7r///tLaxo0bw2NTU1xT2yKnpKYtR1JtwxMnToT1aOpx6u9+woQJYb3qlOk6lbXeRvKa/S8k/ZWk9WZ2uqn6U0n3Sfqlmf1I0lZJcaMaQKOSYXf3/5JU9l/k99o7HAB14XRZIBOEHcgEYQcyQdiBTBB2IBMdneLa09PjUV83NVU06n3u37+/tCZJfX19YT3VN416vlX6vVK655s6RyDqZad6+MeOHQvrVUW/79Ryzampwam/lyq/s5SqY6sTS0kDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJrlpKOjWHOOqlp5YVrjove/r06aW1/v7+0tpI9Pb2hvXUls11XndqGetDhw6F9SpzylNGjYofq6rMKW/6/IQq6LMDmSPsQCYIO5AJwg5kgrADmSDsQCYIO5CJruqzA6iOPjuQOcIOZIKwA5kg7EAmCDuQCcIOZIKwA5lIht3MzjOz35rZRjP7wMx+Uly+3My2m9m64uOq+ocLoFXJk2rMbLqk6e7+npmdJeldSUs1uB/7QXd/cMQ3xkk1QO3KTqoZyf7s/ZL6i68PmNmHkma0d3gA6vYHvWY3s1mSviPp7eKi28zsfTN7zMwmlxyzzMzWmtnaSiMFUMmIz403sz5J/ynpXnd/xsymSdojySX9gwaf6t+cuA6exgM1K3saP6Kwm9kZkl6U9Gt3/6dh6rMkvejuf5K4HsIO1KzliTA2uDzoo5I+HBr04o27034gaUPVQQKoz0jejV8g6XVJ6yWdXpv3p5JukDRXg0/jP5V0S/FmXnRdPLIDNav0NL5dCDtQP+azA5kj7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZIKwA5kg7EAmCDuQCcIOZIKwA5kg7EAmkgtOttkeSVuHfD+1uKwbdevYunVcEmNrVTvHdkFZoaPz2b9242Zr3X1eYwMIdOvYunVcEmNrVafGxtN4IBOEHchE02Ff0fDtR7p1bN06LomxtaojY2v0NTuAzmn6kR1AhxB2IBONhN3MrjSz/zWzj8zsribGUMbMPjWz9cU21I3uT1fsobfLzDYMuWyKmb1iZpuLz8PusdfQ2LpiG+9gm/FG77umtz/v+Gt2M+uRtEnSX0raJukdSTe4+8aODqSEmX0qaZ67N34ChpktlHRQ0qrTW2uZ2T9K2uvu9xX/UU5297/rkrEt1x+4jXdNYyvbZvyv1eB9187tz1vRxCP7fEkfufsWdz8u6WlJSxoYR9dz99ck7f3KxUskrSy+XqnBP5aOKxlbV3D3fnd/r/j6gKTT24w3et8F4+qIJsI+Q9Lvhny/Td2137tL+o2ZvWtmy5oezDCmDdlma4ekaU0OZhjJbbw76SvbjHfNfdfK9udV8Qbd1y1w9z+T9H1JPy6ernYlH3wN1k29059J+rYG9wDsl/RQk4MpthlfI+l2d98/tNbkfTfMuDpyvzUR9u2Szhvy/czisq7g7tuLz7skPavBlx3dZOfpHXSLz7saHs+X3H2nuw+4+ylJP1eD912xzfgaSb9w92eKixu/74YbV6futybC/o6k2Wb2LTM7U9IPJT3fwDi+xszGF2+cyMzGS7pC3bcV9fOSbiq+vknSrxocy+/plm28y7YZV8P3XePbn7t7xz8kXaXBd+Q/lvT3TYyhZFwXSvrv4uODpscm6SkNPq07ocH3Nn4k6WxJr0raLOk/JE3porH9mwa39n5fg8Ga3tDYFmjwKfr7ktYVH1c1fd8F4+rI/cbpskAmeIMOyARhBzJB2IFMEHYgE4QdyARhBzJB2IFM/B+tIjCppYWKvAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pkgutil import get_data\n",
    "import onnx.numpy_helper as nph\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "raw_i = get_data(\"finn\", \"data/onnx/mnist-conv/test_data_set_0/input_0.pb\")\n",
    "x = nph.to_array(onnx.load_tensor_from_string(raw_i))\n",
    "plt.imshow(x.reshape(28,28), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that we partitioned our original network into a parent graph that contained the non-synthesizable nodes and a child graph that contained the bulk of the network, which we turned into a bitfile. We'll load up the parent graph, modify the `StreamingDataflowPartition` node so that it points to the deployed ONNX graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_model = ModelWrapper(build_dir+\"/tfc_w1_a1_dataflow_parent.onnx\")\n",
    "sdp_node = parent_model.graph.node[2]\n",
    "remote_exec_model = build_dir + \"/tfc_w1_a1_pynq_deploy.onnx\"\n",
    "getCustomOp(sdp_node).set_nodeattr(\"model\", remote_exec_model)\n",
    "parent_model.save(build_dir+\"/tfc_w1_a1_dataflow_parent_with_remote_bitfile_exec.onnx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can call `execute_onnx` on the parent graph, which will internally call remote execution with the bitfile once the `StreamingDataflowPartition` node is reached, grab the results, then continue executing the last portion of the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from finn.core.onnx_exec import execute_onnx\n",
    "iname = parent_model.graph.input[0].name\n",
    "oname = parent_model.graph.output[0].name\n",
    "ishape = parent_model.get_tensor_shape(iname)\n",
    "input_dict = {iname: x.reshape(ishape)}\n",
    "ret = execute_onnx(parent_model, input_dict, True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll pass the output of the network through a softmax function to interpret it as probabilities, and plot the per-class probabilities as a bar chart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BarContainer object of 10 artists>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAMp0lEQVR4nO3cf6zdd13H8eeL1qoMgia7f2jbcRttMA2iI9cyJUHDZtJlpjVhJl0CYQbSmFCZQqKdmv1R/4Fhpv7RGJoxQxQsOPnj4qrVCP7hHyy9+xGgq43XOtdWDHeAYDRaGt7+0VNyvLvt/XY79572fZ+PZMn5fr+f3O/7bN0z336/95xUFZKkm9+rpj2AJGkyDLokNWHQJakJgy5JTRh0SWpi87ROfOutt9bs7Oy0Ti9JN6WnnnrqxaqaWenY1II+OzvLwsLCtE4vSTelJP96tWPecpGkJgy6JDVh0CWpCYMuSU0MCnqSPUnOJFlMcmiF4/cnWUry7Oif905+VEnStaz6Wy5JNgFHgJ8HzgMnk8xX1XPLln6qqg6uwYySpAGGXKHvBhar6mxVXQSOAfvWdixJ0vUaEvStwLmx7fOjfcu9I8kXkzyeZPtKPyjJgSQLSRaWlpZexriSpKuZ1EPRzwKzVfUm4G+Bj6+0qKqOVtVcVc3NzKz4QSdJ0ss05JOiF4DxK+5to33fVVVfG9t8FHj4lY+m5WYPPbHm53j+Q/es+TkkrY0hV+gngZ1JdiTZAuwH5scXJPmhsc29wOnJjShJGmLVK/SqupTkIHAC2AQ8VlWnkhwGFqpqHnh/kr3AJeDrwP1rOLMkaQWDvpyrqo4Dx5fte2js9YPAg5MdTZJ0PfykqCQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgYFPcmeJGeSLCY5dI1170hSSeYmN6IkaYhVg55kE3AEuBvYBdyXZNcK614LPAA8OekhJUmrG3KFvhtYrKqzVXUROAbsW2Hd7wIfBv5ngvNJkgYaEvStwLmx7fOjfd+V5M3A9qp6YoKzSZKuwyt+KJrkVcAjwAcHrD2QZCHJwtLS0is9tSRpzJCgXwC2j21vG+274rXAG4G/T/I8cAcwv9KD0ao6WlVzVTU3MzPz8qeWJL3EkKCfBHYm2ZFkC7AfmL9ysKq+WVW3VtVsVc0CXwD2VtXCmkwsSVrRqkGvqkvAQeAEcBr4dFWdSnI4yd61HlCSNMzmIYuq6jhwfNm+h66y9ude+ViSpOvlJ0UlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpiUFBT7InyZkki0kOrXD8V5J8KcmzSf4hya7JjypJupZVg55kE3AEuBvYBdy3QrA/WVU/XlU/CTwMPDLxSSVJ1zTkCn03sFhVZ6vqInAM2De+oKq+NbZ5C1CTG1GSNMTmAWu2AufGts8Db1m+KMn7gA8AW4C3r/SDkhwADgDcdttt1zurJOkaJvZQtKqOVNWPAL8J/M5V1hytqrmqmpuZmZnUqSVJDAv6BWD72Pa20b6rOQb84isZSpJ0/YYE/SSwM8mOJFuA/cD8+IIkO8c27wH+aXIjSpKGWPUeelVdSnIQOAFsAh6rqlNJDgMLVTUPHExyF/Bt4BvAu9dyaEnSSw15KEpVHQeOL9v30NjrByY8lyTpOvlJUUlqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWpiUNCT7ElyJslikkMrHP9AkueSfDHJ3yV5/eRHlSRdy6pBT7IJOALcDewC7kuya9myZ4C5qnoT8Djw8KQHlSRd25Ar9N3AYlWdraqLwDFg3/iCqvp8Vf33aPMLwLbJjilJWs2QoG8Fzo1tnx/tu5r3AH+10oEkB5IsJFlYWloaPqUkaVUTfSia5J3AHPCRlY5X1dGqmququZmZmUmeWpI2vM0D1lwAto9tbxvt+3+S3AX8NvCzVfW/kxlPkjTUkCv0k8DOJDuSbAH2A/PjC5LcDnwU2FtVX538mJKk1awa9Kq6BBwETgCngU9X1akkh5PsHS37CPAa4M+TPJtk/io/TpK0RobccqGqjgPHl+17aOz1XROeS5J0nfykqCQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDUxKOhJ9iQ5k2QxyaEVjr8tydNJLiW5d/JjSpJWs2rQk2wCjgB3A7uA+5LsWrbsBeB+4JOTHlCSNMzmAWt2A4tVdRYgyTFgH/DclQVV9fzo2HfWYEZJ0gBDbrlsBc6NbZ8f7btuSQ4kWUiysLS09HJ+hCTpKtb1oWhVHa2quaqam5mZWc9TS1J7Q4J+Adg+tr1ttE+SdAMZEvSTwM4kO5JsAfYD82s7liTpeq0a9Kq6BBwETgCngU9X1akkh5PsBUjyU0nOA78EfDTJqbUcWpL0UkN+y4WqOg4cX7bvobHXJ7l8K0aSNCV+UlSSmjDoktSEQZekJgy6JDUx6KGoJK2n2UNPrPk5nv/QPWt+jvVm0DWI/4NJNz5vuUhSEzflFbpXi5L0Ul6hS1ITBl2SmjDoktTETXkPXdLa81nVzceg66aw1nExLOrAWy6S1IRBl6QmvOUi3cC81aTrYdClVRhV3Sy85SJJTRh0SWrCoEtSE95Dv05+2ELSjcqgS9KYm/mizVsuktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNTEo6En2JDmTZDHJoRWOf2+ST42OP5lkdtKDSpKubdWgJ9kEHAHuBnYB9yXZtWzZe4BvVNWPAr8PfHjSg0qSrm3IFfpuYLGqzlbVReAYsG/Zmn3Ax0evHwfuTJLJjSlJWk2q6toLknuBPVX13tH2u4C3VNXBsTVfHq05P9r+59GaF5f9rAPAgdHmG4Azk3ojA9wKvLjqqn583xuL77u/11fVzEoH1vX70KvqKHB0Pc95RZKFqpqbxrmnyfe9sfi+N7Yht1wuANvHtreN9q24Jslm4HXA1yYxoCRpmCFBPwnsTLIjyRZgPzC/bM088O7R63uBz9Vq93IkSRO16i2XqrqU5CBwAtgEPFZVp5IcBhaqah74GPAnSRaBr3M5+jeaqdzquQH4vjcW3/cGtupDUUnSzcFPikpSEwZdkppoH/TVvragoyTbk3w+yXNJTiV5YNozrackm5I8k+Qvpz3LekryA0keT/KPSU4n+elpz7Qekvz66M/5l5P8WZLvm/ZM09I66AO/tqCjS8AHq2oXcAfwvg3yvq94ADg97SGm4A+Bv66qHwN+gg3w7yDJVuD9wFxVvZHLv7hxI/5SxrpoHXSGfW1BO1X1lap6evT6P7n8P/bW6U61PpJsA+4BHp32LOspyeuAt3H5N86oqotV9R/TnWrdbAa+f/QZmFcD/zbleaame9C3AufGts+zQcJ2xeibL28HnpzuJOvmD4DfAL4z7UHW2Q5gCfjj0e2mR5PcMu2h1lpVXQB+D3gB+Arwzar6m+lONT3dg76hJXkN8BfAr1XVt6Y9z1pL8gvAV6vqqWnPMgWbgTcDf1RVtwP/BbR/ZpTkB7n8t+4dwA8DtyR553Snmp7uQR/ytQUtJfkeLsf8E1X1mWnPs07eCuxN8jyXb6+9PcmfTnekdXMeOF9VV/4m9jiXA9/dXcC/VNVSVX0b+AzwM1OeaWq6B33I1xa0M/rq4o8Bp6vqkWnPs16q6sGq2lZVs1z+b/25qtoQV2tV9e/AuSRvGO26E3huiiOtlxeAO5K8evTn/k42wMPgq1nXb1tcb1f72oIpj7Ue3gq8C/hSkmdH+36rqo5PcSatvV8FPjG6eDkL/PKU51lzVfVkkseBp7n8213PsIG/BsCP/ktSE91vuUjShmHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUxP8B9uoCk0KMtNwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def softmax(x):\n",
    "    \"\"\"Compute softmax values for each sets of scores in x.\"\"\"\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum()\n",
    "\n",
    "logits = ret[oname].flatten()\n",
    "prob = softmax(logits)\n",
    "\n",
    "plt.bar(np.arange(10), prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the network correctly predicts this as a digit 2 with high probability. This concludes our tutorial on how to take a simple fully-connected BNN all the way down to hardware with FINN, and execute it remotely on a PYNQ board."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Throughput Test on PYNQ Board <a id='throughput'></a>\n",
    "In addition to the functional verification, FINN also offers the possibility to measure the network performance directly on the PYNQ board. This can be done using the core function `throughput_test`. In the next section we import the function and execute it.\n",
    "First we extract the `remote_exec_model` again and pass it to the function. The function returns the metrics of the network as dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network metrics:\n",
      "runtime[ms]: 1.4772415161132812\n",
      "throughput[images/s]: 676937.378954164\n",
      "DRAM_in_bandwidth[Mb/s]: 75.81698644286635\n",
      "DRAM_out_bandwidth[Mb/s]: 27.07749515816656\n"
     ]
    }
   ],
   "source": [
    "from finn.core.throughput_test import throughput_test\n",
    "\n",
    "child_model = ModelWrapper(getCustomOp(sdp_node).get_nodeattr(\"model\"))\n",
    "res = throughput_test(child_model)\n",
    "print(\"Network metrics:\")\n",
    "for key in res:\n",
    "    print(str(key) + \": \" + str(res[key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Together with the values for folding we can evaluate the performance of our accelerator. Each layer has a total folding factor of 64 and because the network is fully pipelined, it follows: `II = 64`. II is the initiation interval and indicates how many cycles are needed for one input to be processed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We reach approximately 43% of the ideal performance.\n"
     ]
    }
   ],
   "source": [
    "II = 64\n",
    "# frequency in MHz\n",
    "f_MHz = 100\n",
    "# expected throughput in MFPS\n",
    "expected_throughput = f_MHz / II\n",
    "# measured throughput (FPS) from throughput test, converted to MFPS\n",
    "measured_throughput = res[\"throughput[images/s]\"] * 0.000001\n",
    "# peformance\n",
    "print(\"We reach approximately \" + str(round((measured_throughput / expected_throughput)*100)) + \"% of the ideal performance.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The measured values were recorded with a batch size of 1000 and at a frequency of 100 MHz. We will be improving the efficiency of the generated accelerator examples in the coming FINN releases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
